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Abstract 


This in-brief technical report documents the results from two different analytic approaches for 
examining the reliability of the slope for easyCBM” reading measures in Grades K-8. Results 
varied by grade, assessment measure, and the analytic approach. Results patterns are discussed. 
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In-Brief: Reliability of the Slope of the eayyCBM® Reading Measures 
Background 


The National Center on Intensive Interventions (NCI; https://intensiveintervention.org) 
evaluates both screening and progress monitoring tools for their technical adequacy, with the 
goal of helping educators select appropriate tools to meet their needs. Many of the analyses 
required by NCII are already part of our standard practice in developing assessments and are thus 
described in detail in the many technical reports we publish as part of our assessment 
development process. Other analyses, such as the reliability of the slope, reported here, fall 
outside the scope of our standard technical reports. In this brief, we present the results of our 
analyses of the reliability of the slope conducted with a population of students in need of 
intensive intervention. 


Methods 


Sample 

The analytic sample consisted of students who took easyCBM” reading progress monitoring 
measures during the 2014-2015, 2015-2016, and 2016-2017 school years. All students in the 
sample were identified by their districts as needing intensive intervention in the specific skill 
area targeted by the assessments for which their data were included in this study. Data from this 
study are a subset of a much larger extant data set. The larger data set includes scores for all 
students in all districts with easyCBM" accounts covering fall of 2014 to spring of 2017. From 
this larger data set, we included only those students identified as needing intensive intervention 
who had a minimum of 10 assessment scores for a given assessment measure with a minimum of 
20 weeks between the first and last administration occasion. Thus, sample sizes varied by grade 
and assessment measure administered. 


Analyses 
We analyzed the reliability of the slope using two approaches, Pearson split-test correlation 
analysis and reliability of the slope. 


Pearson split-test correlation analysis. For each student, assessments were divided into 
two data subsets comprised of odd and even numbered tests, respectively, depending on the 
chronological order in which they were taken. An OLS slope of improvement (growth) was 
estimated for each data subset and for each student. Table 1, below, summarizes Pearson 
correlation coefficients as a measure of the strength of association between even and odd 
numbered test slopes. 


Reliability of slope. Reliability of the slope is defined here as the ratio of the true score 
variance to the total variance. The true score variance is the random slope variance in a mixed- 
effects growth model (Ime4 package; Bates, Maechler, Bolker, & Walker, 2015) in the R 
software environment (R Core Team, 2018). The total variance is the estimation of total variance 
of each student’s individual slope of improvement (R Core Team, 2018). Table 2 summarizes 
correlation coefficients as a measure of the strength of association between the true score 
variance and the total variance. 
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Results 
Results varied by measure and grade, as well as by analytic approach. Table 1 presents the 
results of the Pearson split-test correlation analyses and Table 2 presents the results of the 
reliability of the slope analyses. 


Table 1 
Pearson Split-test Correlation Analyses Results 
Correlation 95% Confidence Interval 
Measure Grade n ; 
coefficient (7) Lower Upper 
Trettenamice K 253 83" 79 87 
(LN) 1 44 lee 53 83 
Phoneme K 42 79" 64 88 
Segmenting (SEG) 1 91 oe 61 81 
ieee Sounds K 237 84" 79 87 
(LS) 1 199 60— 50 68 
K 10 81 38 96 
Word Reading l 348 88" 85 90 
Fluency (WRF) 2 155 42 28 54 
3 60 42 19 61 
1 75 85. ag 90 
2 412 33 24 Al 
3 387 42 33 49 
Passage Reading 4 163 26 ll 40 
Fluency (PRF) 5 263 25 13 36 
6 64 40 18 59 
7 11 24 -.68 Ril 
8 137 29 13 43 
p) 4l 40 ll 63 
3 7 .00 -.76 75 
4 LD) bal 22 58 
aan 5 ib 66 36 86 
6 7 56 -.33 92 
fi NA NA NA 
8 32 02 -.33 36 


Note. Lower bound of the confidence interval around the median correlation estimate falls 
below 0.50 but meets or exceeds 0.40. 
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Table 1 cont. 
Pearson Split-test Correlation Analyses Results 


Correlation 95% Confidence Interval 
Measure Grade n ‘ 
coefficient (7) Lower Upper 

2 138 34 18 48 

3 aly -.12 -.29 07 
Multiple Choice 4 42 -.05 235 26 
eae aoe 5 116 17 -.02 34 
(MCRC) 6 NA NA NA NA 

7 16 09 -.43 56 

8 238 14 01 .26 

i) 9 34 -.42 .82 

4 4 -.46 -.99 .90 
CCSS Reading 5 16 .26 -.27 .67 
(CCSS) 6 -.54 -.94 49 

f] 4 a da -.94 98 

8 22 -.14 -.53 30 


Note. NA = Not analyzed. © Lower bound of the confidence interval around the median 
correlation estimate falls below 0.50 but meets or exceeds 0.40. 
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Table 2 


Reliability of Slope Analyses Results 


Correlation Confidence Interval 
Measure Grade n coetiicient ©) ae Gee: 
(eeter Names K 253 ‘oe 74 1.00 
(LN) 1 44 719° 50 1.00 
Phoneme K 42 83" 53 1.00 
Segmenting (SEG) 1 91 om 51 98 
Teeter Sounds K 237 86 aE) 1.00 
(LS) 1 199 be 57 89 
K 10 1.00" 45 1.00 
Word Reading 1 348 92” 81 1.00 
Fluency (WRF) 2 155 bv 58 94 
3 60 54 29 87 
1 75 83 65 1.00 
D 412 62 52 74 
3 387 56 45 68 
Passage Reading 4 163 ae, eu he 
Fluency (PRF) 5 263 42 29 56 
6 64 48 23 81 
7 iG 03 NA NA 
8 137 40 oy) 62 
2 41 07 00 42 
3 7 AT 00 1.00 
4 oa) 48 08 1.00 
eat 5 17 a 33 1.00 
6 es 44 00 1.00 
7 1 NA NA NA 
8 32 08 00 55 


Note. NA = Not analyzed. ‘Lower bound of the confidence interval around the median 
correlation estimate falls below 0.50 but meets or exceeds 0.40. 
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Table 2 cont. 
Reliability of Slope Analyses Results 


Correlation Confidence Interval 
Measure Grade n ; 
coefficient (7) Lower Upper 
2 138 37 2) 7 
5 117 .20 02 41 
Multiple Choice 4 42 28 00 510) 
irene 5 116 A4 26 65 
(MCRC) 6 NA NA NA NA 
if 16 16 .00 wi) 
8 238 oak 12 43 
3 9 2 nel 1.00 
4 4 .08 .00 1.00 
CCSS Reading i) 16 ay .00 1.00 
(CCSS) 6 23 00 1.00 
i 67 NA NA 
8 22 .06 .00 58 


Note. NA = Not analyzed. ‘Lower bound of the confidence interval around the median 
correlation estimate falls below 0.50 but meets or exceeds 0.40. 


Discussion 


Results varied by measure, grade, and analytic approach, with clear patterns emerging. We 
document reasonable reliability of the slope, whereby the lower bound of the 95% confidence 
interval around the median correlation estimate falls below 0.50 but meets or exceeds 0.40, for 
the early/emergent literacy measures in Grades K-1 using both analytic approaches (LN, SEG, 
and LS) and additionally for WRF and PRF using the reliability of slope approach in Grades K-2 
and Grades 1-3, respectively. Results for later elementary grades (i.e., generally, Grade 4 and up) 
and three measures (i.e., VOC, MCRC, and CCSS) were relatively less encouraging. Spanish 
language measures for all available grades were not analyzable due to insufficient sample size. 
Given extant data were used for the analyses reported here, we are currently planning a series of 
studies to better control for sample population characteristics and size and anticipate improved 
reliability of the slope results for these later grades and measures. 
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